Method of streaming multimedia data

ABSTRACT

The invention relates to a method of streaming multimedia for providing seamless stream switching. The method consists in streaming multimedia data from a server ( 15 ) to a client ( 14 ) over a network ( 13 ) having a variable bandwidth. The client is intended to decode a subset of streams within a set of streams having various predetermined bit rates. The decoded subset of streams has bit rates compatible with the network bandwidth. The method comprises the steps of configuring the client so that the client can decode all the streams within the set of streams, playing all the streams within the set of streams and muting all the streams within the set of streams, except the subset of streams.

FIELD OF THE INVENTION

The present invention relates to a method of streaming multimedia datafrom a server to a client over a network having a variable bandwidth,the client being intended to decode a subset of streams within a set ofstreams having various predetermined bit rates, said subset of streamshaving bit rates compatible with the network bandwidth.

The present invention also relates to a server and a client for carryingout this method, a telecommunication system implementing this method anda computer program for carrying out this method.

The present invention is particularly relevant for systems of multimediastreaming over networks whose effective instantaneous bandwidth may varywith time. For example, it applies to transmissions over mobiletelephony systems using various transmission standards such as MPEG4(Moving Expert Picture Group-4) or H263.

BACKGROUND OF THE INVENTION

Multimedia networks such as the GPRS (General Packet Radio System) orUMTS (Universal Mobile Telephony System) and the Internet do not providea guaranteed quality of service. The effective instantaneous bandwidthvaries with time because of congestions, transmission errors or resourcesharing for instance. In the framework of multimedia streaming over suchnetworks, adaptation mechanisms to these variations have to be providedin order to minimize data losses on the one hand and maximize thequality of the content received by the client on the other hand.

A classical approach to this problem is known as “Stream Switching”. Thesame content is encoded by a media encoder into a set of streams atseveral bit rates. With respect to the currently available bandwidth, anadequate stream, i.e. having an adequate bit rate with respect to theavailable bandwidth, is selected to be transmitted to the client via amedia server, which operates the corresponding stream switching, forselecting said adequate stream to be sent.

The way a stream switching is usually performed is describedhereinafter. Based on the bit rates of the various streams and on theavailable bandwidth, the server selects a stream having a bit ratecompatible with the available bandwidth. The server can also select asubset of streams, when the client is intended to decode simultaneouslymany streams, e.g. an audio and a video stream. In the following, a“subset of streams” shall designate one stream or a few streams. If theselected subset of streams differs from the subset of streams currentlydecoded by the client, a stream switching is performed.

In order to perform such a stream switching, information concerning theselected subset of streams is communicated to the client, so that theclient can change its configuration in order to be able to decode thisselected subset of streams, with respect to the content and bit rates ofthis selected subset of streams.

Thus, when a server switches from one subset of streams to another one,e.g. in order to adapt the bit rate of the delivered subset of streamsto the available bandwidth of the network, a new decoder configurationcorresponding to the new delivered subset of streams has to be sent tothe client decoder. The decoder is then reinitialised with the newdecoder configuration. The stream switching is therefore not seamlessfor the client and may impact the service quality from the end user'spoint of view.

For example, suppose one wants to perform a stream switching using theIETF (Internet Engineering Task Force) standard, which is RTSP (RealTime Streaming Protocol) described in “Real Time Streaming Protocol(RTSP), RFC2326, April 1998” written by Schulzrinne H. et al. Thefollowing commands have to be sent by the client to the server:

-   -   TEARDOWN: to stop the current subset of streams    -   DESCRIBE: to get information on the new subset of streams    -   SETUP: to configure the new session, ie to indicate to the        server that the decoder has been reinitialised    -   PLAY: to start streaming the new subset of streams in the new        session.        These commands and the answer to these commands require a long        time, for example a few seconds. Actually, each request from the        client to the server is followed by an acknowledgment sent by        the server to the client, which means that each command requires        a whole round trip over the network. The service is therefore        interrupted on the end user side during a few seconds, which is        not acceptable.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a method of streamingmultimedia data, which leads to a seamless stream switching.

To this end, a method of streaming multimedia data according to theinvention, as described in the opening paragraph, is characterised inthat it comprises the steps of configuring the client so that the clientcan decode all the streams within the set of streams, playing all thestreams within the set of streams and muting all the streams within theset of streams, except the subset of streams.

According to the invention, the client decoder is first configured inorder to be able to decode all streams within the set of streams. Then,all streams within the set of streams are played, but a large number ofthese streams are muted, i.e. they do not reach the final decoder inorder to be decoded. Only the subset of streams compatible with thenetwork bandwidth is not muted, i.e. reaches the client decoder to bedecoded. When a stream switching is to be performed between a currentlydecoded subset of streams and a next subset of streams, the currentlydecoded subset of streams is muted and the next subset of streams issimply unmuted. Such a stream switching thus only requires muting andunmuting two subsets of streams, and does not require any configurationof a new session. Such a stream switching is therefore seamless.

Advantageously, the step of muting all the streams except the subset ofstreams is performed by the server on a request from the client inaccordance with the MUTE/UNMUTE extension of the Real Time StreamingProtocol. The method uses standard IETF protocols so that servers andclients implemented by different parties can interoperate.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described in more detail, by way of example,with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a telecommunication systemcomprising a server and a client according to the invention;

FIG. 2 is a block diagram illustrating a telecommunication system with aclient according to a first embodiment of the invention;

FIG. 3 is a block diagram illustrating a telecommunication system with aclient according to a second embodiment of the invention;

FIG. 4 is a diagram for illustrating a method of streaming and a streamswitching according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

A telecommunication system according to the invention is depicted inFIG. 1. Such a telecommunication system comprises a server 15 comprisingplaying means 11 and muting means 12, a network 13 and a client 14. Thenetwork 13 is, for example, a mobile radiotelephony network or theInternet network.

The server 15 receives a set of streams {S1, S2, S3, S4}. These streamscorrespond to a multimedia content produced by a multimedia source,after encoding. For a given multimedia source, all the produced encodedstreams within the set {S1, S2, S3, S4} comprise the same multimediacontent, but have different predetermined bit rates. The bit rates arepredetermined in order for the system to be able to provide the client14 with a set of various qualities of services, selected on the one handwith respect to the nature of the transmitted content, each content(audio, fixed images, video, etc.) requiring a specific availablebandwidth, and on the other hand with respect to the quality of servicerequired by the end user on the client side.

The various available bit rates are also useful to enable the system tocope with variations of the available bandwidth, which often cause areduction of the bit rate of the transmitted streams. As a matter offact, the network's effective bandwidth may vary with time with respectto several parameters. These parameters include current traffic andradio transmission conditions, which may considerably affect the currentavailable bandwidth. With respect to this current available bandwidthand the required quality of service, the adequate stream has to bedelivered to the client 13.

The method of streaming multimedia data according to the invention isdescribed hereinafter. When the client 14 desires to decode a multimediacontent, it first asks the server 15 for information about the variousstreams S1 to S4 comprising this multimedia content. This informationcomprises the bit rates of each stream S1 to S4. Using this information,the client 13 is configured so that it can decode all the streams withinthe set of streams {S1, S2, S3, S4}. How such a configuration isperformed is described in FIGS. 2 and 3. Then, according to theavailable network bandwidth, a subset of streams is selected, which isto be decoded by the client 14.

The measure of the available bandwidth can be performed by the server15. In this case, the server 15 will select the subset of streams to bedecoded by the client 14. This is particularly advantageous when theserver 15 has information about the client 14, for example the variousdecoders available in client 14. The measure of the available bandwidthcan also be performed by the client 14. In this case, the client 14 willselect the subset of streams to be decoded. This would be the case, forexample, in a mobile radiotelephony network, where a lot of differentclients are linked to a same server. Actually, in such a network, theserver 15 cannot have information about all the clients in the network.The available bandwidth can also be measured by the server 15 and sentto the client 14, which will then select the subset of streams to bedecoded.

Suppose the subset of streams to be decoded by the client 14 is thestream S1. Then all the streams within the set of streams {S1, S2, S3,S4} are played by the server 15, thanks to the playing means 11, but thestreams S2, S3 and S4 are muted, thanks to the muting means 12. Playingand muting a stream means that the stream is virtually played by theplaying means 11, but the stream does not reach the client 14 and thevirtual play does not consume any resource on the server side. Forexample, suppose a stream comprises 10 stream elements enumerated 1 to10, the first stream element 1 being played at a first time t1, thesecond stream element 2 at a time t2 and so on, and suppose the streamis muted at time t4 and unmuted at time t8. Then the client 14 receivesthe stream elements 1 to 3 and, after a time equal to t8−t4, the client14 receives the stream elements 8 to 10.

As the streams S2, S3 and S4 are muted, the client 14 only receives thestream S1 and can decode this stream because it has been configured inorder to be able to decode all the streams. Suppose the availablebandwidth changes and a stream switching is to be performed, for examplethe stream S2 is selected to be decoded by the client 14. In order toperform such a stream switching, the stream S1 is muted and the streamS2 is unmuted, so that only the stream S2 reaches the client 14, whichstream S2 can be decoded by the client 14 because it has been configuredin order to be able to decode all the streams.

The method in accordance with the invention thus allows performing aseamless stream switching. Actually, the time required to mute a streamand unmute another stream on the server side is very small, so thatthere is no service interruption on the client side, which could benoticed by an end user. The end user on the client side might be awarethat a stream switching has been performed, because the quality of thecurrently decoded stream might have changed. Compared to a streamswitching in accordance with the state of the art, the stream switchingperformed by the method in accordance with the invention does notrequire configuring a new session each time a new stream is selected tobe decoded by the client 14, because a session is already establishedfor each stream within the set of streams {S1, S2, S3, S4}, as thesestreams are played by the playing means 11. As a consequence, the streamswitching according to the invention is much more seamless than a streamswitching according to the state of the art.

It is important to notice than the muting means 12 might be separatedfrom the server 15. For example, the muting means 12 might be a separatedevice placed between the server 15 and the client 13, or might formpart of the client 14.

FIG. 2 illustrates a telecommunication system according to a firstembodiment of the invention. This telecommunication system comprises theserver 15, the network 13 and the client 14. The client 14 comprises acontroller 21 and ten decoders D1 to D10.

In the following example, a multimedia content to be decoded by theclient 14 has been encoded into 10 streams S1 to S10, forming a set ofstreams {S1-S10}. The five first streams S1 to S5 contain a videocontent and the five last streams S6 to S10 contain an audio content.Playing the multimedia content consists in playing one video stream andone audio stream. Examples of streams S1 to S10 are given in the 2following tables, which principally define the formats of the streamsand their bit rates. Stream Format Size Profile Frames/sec Bit Rate S1H263 QCIF Single 2 25 S2 MPEG-4 QCIF Single 10 50 video S3 MPEG-4 QCIFSingle 25 100 video S4 MPEG-4 CIF Single 25 200 video S5 MPEG-4 CIFAdvanced 25 400 video simple Stream Format Bit Rate S6 AdaptiveMulti-Rate 5 S7 Code Excited Linear Prediction 10 S8 Advanced AudioCoding mono 32 S9 Advanced Audio Coding stereo 64 S10 Advanced AudioCoding + advanced tools 128

When the client 14 desires to decode the multimedia content, thecontroller 21 asks for information about each stream S1 to S10. Forexample, the multimedia content might be contained in a file, which hasan URL address (URL stands for Universal Resource Locator). Knowing theaddress of the content, the controller 21 asks for a description of thestreams comprising this multimedia content, by sending a request to theserver 15 via the network 13. For example, if the RTSP protocol is used,such a request is a “DESCRIBE” command which has the followingstructure:

-   -   DESCRIBE rtsp://movie RTSP/1.0    -   CSeq: 1.

In this command, the URL address of the multimedia content is“rtsp://movie”, the protocol used is “RTSP/1.0” and the sequence numberCseq of this message is 1.

The server 15 then sends the descriptions of the streams comprising thedesired multimedia content. The messages sent by the server 15 to theclient 14 have the following structure:

-   -   RTSP/1.0 200 OK    -   CSeq: 1    -   Content-Type: application/sdp    -   Content-Length: xxx    -   a=control:rtsp://movie    -   T=0 0    -   m=video 7722 RTP/AVP 96    -   a=rtpmap:96 MP4V-ES/1000    -   a=control:rtsp://movie/video1    -   a=br:400    -   m=audio 7724 RTP/AVP 97    -   a=rtpmap:97 mpeg4-generic/44100/2    -   a=control:rtsp://movie/audio1    -   a=br:128    -   m=video 7726 RTP/AVP 98    -   a=rtpmap:98 MP4V-ES/1000    -   a=control:rtsp://movie/video2    -   a=br:200    -   m=audio 7724 RTP/AVP 99    -   a=rtpmap:99 mpeg4-generic/44100/2    -   a=control:rtsp://movie/audio2    -   a=br:68

These messages correspond to a description of the streams S5, S10, S4and S9, respectively.

As the description of a given stream comprises the format of the streamand its bit rate, the controller can configure one of its decoders sothat this decoder is able to decode the given stream. In the example ofFIG. 2, the decoder D1 is configured to decode the stream S1, thedecoder D2 is configured to decode the stream S2 and so on.

The client 14 and the server 15 also have to agree on transportparameters, i.e. UDP port numbers. This is effected thanks to a SETUPcommand, which has the following structure, given for the stream S10:

-   -   SETUP rtsp://movie/audio1 RTSP/1.0    -   CSeq: 2    -   Transport: RTP/AVP;unicast;client_port=8000-8001

The server then answers this message by sending a message having thefollowing structure:

-   -   RTSP/1.0 200 OK    -   CSeq: 2    -   Transport: RTP/AVP;unicast;client_port=8000-8001;        server_port=9000-9001    -   Session: 12345678

In the following, we suppose that the measure of the bandwidth isperformed by the client 14, e.g. by the controller 21, and that theclient 14 selects the subset of streams compatible with the availablebandwidth.

Once the decoders of the client 14 have been configured, a subset ofstreams to be decoded by the client 14 is selected, for example thestreams S5 and S10.

Then the controller 21 requests that the streams S1 to S4 and S6 to S9are muted. Such a request is sent via the network 13 to the server 15,which comprises muting means. If the RTSP protocol is used, such arequest uses the MUTE extension of this protocol, and has the followingstructure, given for the streams S9 and S4:

-   -   MUTE rtsp://movie/audio2 RTSP/1.0    -   CSeq: 8    -   Session: 12345678    -   MUTE rtsp://movie/video2 RTSP/1.0    -   CSeq: 9    -   Session: 12345678

The streams S1 to S10 are then played by the server 15. For example, theclient 14 can request that the streams S1 to S10 are played. If the RTSPprotocol is used, such a request uses the PLAY command and has thefollowing structure:

-   -   PLAY rtsp://movie RTSP/1.0    -   CSeq: 12    -   Range: npt=0−    -   Session: 12345678

The server then sends an acknowledgment, which has the followingstructure:

-   -   RTSP/1.0 200 OK    -   CSeq: 12    -   Session: 12345678

As all streams within the set of streams {S1-S10} are muted, except thestreams S5 and S10, the streams S5 and S10 reaches the controller 21,which sends them to the decoders D5 and D10, respectively. Thecontroller might comprise switches or multiplexers in order to send theincoming streams to the right decoders. The controller 21 might alsocomprise a plurality of ports, each assigned to one decoder, and receivea given stream on the port corresponding to the decoder intended todecode this given stream.

Suppose a stream switching is to be performed because the availablebandwidth has decreased. For example, the client 14 would like to playthe streams S4 and S9, which have lower bit rates than the streams S5and S10, respectively. The controller 21 then requests that the streamsS5 and S10 are muted and that the streams S4 and S9 are unmuted. Such arequest is sent via the network 13 to the server 15. If the RTSPprotocol is used, such a request uses the MUTE/UNMUTE extension of thisprotocol, and has the following structure:

-   -   MUTE rtsp://movie/audio1 RTSP/1.0    -   CSeq: 13    -   Session: 12345678    -   MUTE rtsp:/movie/video1 RTSP/1.0    -   CSeq: 14    -   Session: 12345678    -   UNMUTE rtsp://movie/audio2 RTSP/1.0    -   CSeq: 15    -   Session: 12345678    -   UNMUTE rtsp://movie/video2 RTSP/1.0    -   CSeq: 16    -   Session: 12345678

The server then sends an acknowledgment, which has the followingstructure:

-   -   RTSP/1.0 200 OK    -   CSeq: 13    -   Session: 12345678    -   RTSP/1.0 200 OK    -   CSeq: 14    -   Session: 12345678    -   RTSP/1.0 200 OK    -   CSeq: 15    -   Session: 12345678    -   RTSP/1.0 200 OK    -   CSeq: 16    -   Session: 12345678

Then, when the streams S4 and S9 reach the client 14, the controller 21sends them to the decoders D4 and D9, respectively.

FIG. 3 illustrates a telecommunication system according to a secondembodiment of the invention. The client 14 comprises two decoders D1 andD2, and two memories 31 and 32. The first decoder D1 is able to decodevideo streams and the second decoder D2 is able to decode audio streams.

In this embodiment, the decoders D1 and D2 are configured by means ofconfiguring parameters, which are loaded in the memories 31 and 32. Whenthe server 15 sends a description of each stream within the set ofstreams {S1-S10} via the network 13 to the client 14, the controller 21computes decoding parameters P1 to P10 and loads these parameters intothe memories 31 and 32. The parameters P1 are computed in such a waythat, knowing these parameters, the decoder D1 is able to decode thestream S1. The parameters P2 are computed in such a way that, knowingthese parameters, the decoder D1 is able to decode the stream S2, and soon. The parameters P1 to P5 are loaded into the memory 31 and used toconfigure the decoder D1. The parameters P6 are computed in such a waythat, knowing these parameters, the decoder D2 is able to decode thestream S6, and so on. The parameters P6 to P10 are loaded into thememory 32 and used to configure the decoder D2.

When a given subset of streams is to be decoded by the client 14, as hasbeen described hereinbefore, the configuring parameters corresponding tothis given subset of streams are sent to the decoders D1 and D2 by thecontroller 21, in order to configure these decoders so that they candecode the given subset of streams. Such a configuration might require acertain time. When a stream switching is effected, a new configurationhas to be sent to the decoders, i.e. new configuring parameters have tobe sent to the decoders.

Suppose the stream S1 is decoded by the decoder D1, and a streamswitching is effected in order to decode the stream S2. The decoder D1keeps its configuration corresponding to the parameters P1 until thereis no more data from the stream S1 reaching the decoder D1. At thattime, the stream S2 reaches the decoder D1, but the decoder D1 is notable to decode the stream S2 because the reconfiguration requires acertain time. This might lead to service interruptions. To solve thisproblem, a buffer might be used between the decoder and a displayintended to display the decoded streams. This buffer stores decodedstreams, i.e. audio and/or video, corresponding to a time which is atleast the time required for the reconfiguration of the decoders.

Thanks to this buffer, the end user does not detect any serviceinterruption, because decoded streams are fetched from the buffer anddisplayed while the decoders are being reconfigured.

FIG. 4 illustrates a method of streaming multimedia and a streamswitching according to the invention. Such a method might be implementedby the client 14 of FIGS. 2 and 3.

At step 401, the client 14 desires to decode a multimedia content, whichis, for example, comprised in a file having an URL address. At step 402,the client 14 asks for information about the various streams comprisingthe multimedia content. At step 403, the client is configured in orderto be able to decode all the streams comprising the desired multimediacontent. Such a configuration might be performed as has been describedin FIG. 2 or 3.

At step 404, the client 14 requests that the streams comprising thedesired multimedia content are played and, at step 405, the clientrequests that the streams are muted, except a subset of streams whichthe client desires to decode, i.e. the subset of streams having bitrates compatible with the current available bandwidth. The steps 404 and405 advantageously are simultaneous. At step 406, the unmuted subset ofstreams, which reaches the client 14, is decoded. This step 406 mightrequire a decoder configuration if a client in accordance with thesecond embodiment of the invention is used, as has been described inFIG. 3.

At step 407, the client 14 checks if the bandwidth has changed. If thebandwidth has not changed, the currently decoded subset of streams staysunmuted and the client 14 decodes this subset of streams. If thebandwidth has changed, a subset of streams to be decoded is selectedaccording to the new available bandwidth, at step 408. If this subset ofstreams is different from the currently decoded subset of streams, theclient requests, at step 409, that the currently decoded subset ofstreams is muted and, at step 410, that the new subset of streams to bedecoded is unmuted. The steps 409 and 410 advantageously aresimultaneous. Then, the new subset of streams is decoded at step 406.

The method for streaming multimedia according to the invention can beimplemented in an integrated circuit, which is intended to be integratedin a server or in a client. A set of instructions that is loaded into aprogram memory causes the integrated circuit to carry out the method forstreaming multimedia. The set of instructions may be stored on a datacarrier such as, for example, a disk. The set of instructions can beread from the data carrier so as to load it into the program memory ofthe integrated circuit, which will then fulfil its role.

Any reference sign in the following claims should not be construed aslimiting the claim. It will be obvious that the use of the verb “tocomprise” and its conjugations does not exclude the presence of anyother elements or steps besides those defined in any claim. The word “a”or “an” preceding an element or a step does not exclude the presence ofa plurality of such elements.

1. A method of streaming multimedia data from a server (15) to a client(14) over a network (13) having a variable bandwidth, the client beingintended to decode a subset of streams within a set of streams (S1, S10)having various predetermined bit rates, said subset of streams havingbit rates compatible with the network bandwidth, the method beingfurther characterised in that it comprises the steps of configuring theclient so that the client can decode all the streams within the set ofstreams, playing all the streams within the set of streams and mutingall the streams within the set of streams, except the subset of streams.2. A method of streaming multimedia data according to claim 1, whereinthe step of muting all the streams except the subset of streams isperformed by the server on a request from the client in accordance withthe MUTE/UNMUTE extension of the Real Time Streaming Protocol.
 3. Aserver (15) for serving a client (14) with a subset of streams over anetwork (13) having a variable bandwidth, said server comprising meansfor selecting the subset of streams within a set of streams havingvarious predetermined bit rates, means for playing (11) all the streamswithin the set of streams and means for muting (12) all the streamswithin the set of streams, except the subset of streams.
 4. A server asclaimed in claim 3, wherein the means for selecting the subset ofstreams comprise means for measuring the network bandwidth.
 5. A serveras claimed in claim 3, wherein the means for selecting the subset ofstreams are controlled by a request from the client.
 6. A client (14)intended to decode a subset of streams within a set of streams havingvarious predetermined bit rates, said streams being sent over a network(13) having a variable bandwidth, said client comprising configuringmeans (21) in order to be able to decode all the streams within the setof streams, means for selecting a subset of streams having bit ratescompatible with the network bandwidth and means for muting all thestreams within the set of streams, except the subset of streams.
 7. Atelecommunication system comprising a server for serving a client with asubset of streams, said server comprising means for selecting the subsetof streams within a set of streams having various predetermined bitrates, means for playing all the streams within the set of streams andmeans for muting streams, a network having a variable bandwidth and aclient intended to decode the subset of streams, said client comprisingconfiguring means in order to be able to decode all the streams withinthe set of streams, means for selecting a subset of streams having bitrates compatible with the network bandwidth and means for requestingthat all the streams within the set of streams, except the subset ofstreams, are muted.
 8. A computer program comprising a set ofinstructions which, when loaded into a processor or a computer, causesthe processor or the computer to carry out the method as claimed inclaim 1.